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Introduction and summary 


We have a problem. Increasing teacher and teaching effectiveness is arguably the 
paramount challenge facing public elementary and secondary education, and 
we have too few proven-effective ways of getting this done. Federal funding is 
pouring into initiatives that emphasize measurement and improvement of teacher 
performance, including the Obama administrations signature education-reform 
initiatives like the Race to the Top program for states, the implementation of data 
systems that track student achievement, and funds dedicated to investments in 
innovative models of educational improvements in districts. Yet there is no stock- 
pile of effective teacher professional development and training approaches from 
which states and districts can choose. 

Further, the evidence suggests that most teacher professional development has 
little if any impact, anyway. The gaps between the stated aims of federal and state 
policy, the needs of the teacher workforce, and proven solutions that improve 
teacher and teaching effectiveness are a serious impediment to any effort to 
improve student achievement. In the clutter and clamor of claims and tools for 
improving teachers’ impacts, it is critical that state and district superintendents, 
principals, school boards, and reform leaders grasp the importance of their 
choices and direct their attention to evidence-supported models. 

It is a travesty that despite districts spending thousands of dollars per teacher each 
year on professional development, these dollars are most often spent on models 
that are known to be ineffective. These are predominantly one-time workshops that 
focus mostly on awareness or general knowledge rather than specific skills, or 
models that have little basis in what is known about effective instruction, curricu- 
lum, or classroom interactions. No wonder there is so little evidence of impact. 
Federal and state policymakers should take note — unless evidence standards and 
rigorous review are used to channel money well, most of the current investment 
in teacher professional development is unlikely to have its intended effect. With 
student achievement at an all-time low, particularly in comparison with other 
nations, with education such a critical piece of economic development, and with 
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state and local financial resources strained to the breaking point, it is imperative 
that education leaders use existing money wisely and efficiently. 

We all agree teachers are the critical leverage point for improving a host of desired 
student-learning outcomes — achievement test scores, critical thinking and prob- 
lem solving, teamwork, and life skills. So it seems only sensible to invest in teach- 
ers’ skills and knowledge, an argument the public also endorses. But unless those 
investments are targeted for and deliver proven results for students — real learning 
gains and skills that make them competitive in the workplace — opportunities 
both for the improvement of the public education system and for the millions of 
children it serves will be squandered. 

The aim of this report is to illustrate features of new evidence-supported 
approaches to professional development that have promise for closing not only 
the evidence gap, but the achievement gap as well. The focus is on one web-based, 
scalable approach to professional development — MyTeachingPartner, or MTP — 
that illustrates how evidence- driven professional development can be designed 
and used to improve teacher effectiveness and student learning. MTP uses a 
standardized method of online, individualized coaching and a library of highly 
focused video clips showing effective teachers in action that are tightly coupled 
with a standardized metric for observing teacher practice in the classroom, called 
the Classroom Assessment Scoring System, or CLASS. 

CLASS and MTP are examples of recently developed models of teacher assess- 
ment and professional development funded by grants from a range of federal 
sources. They include models for observing teachers’ instruction in mathematics 
lessons that are useful in modeling feedback about instruction in the upper grades. 
There are now professional-development tools that show promise for improving 
instruction and children’s math skills in preschool . 1 In early literacy, there are now 
videos to provide teachers feedback with demonstrable gains for students’ skills as 
well as statewide models that connect individualized feedback, coursework, and 
assessment of students’ school-readiness skills in a program of teacher profes- 
sional development . 2 And with the recent work of the Gates Foundation-funded 
Measures of Effective Teaching project — spurred by policies that require assess- 
ment of teachers’ performance, observations of teachers’ instruction, and the use 
of video examples of effective practice — all of these approaches will take on a 
much larger role in districts’ professional development approaches. 
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Together, CLASS and MTP reflect an emerging new wave of teacher-performance 
assessment and improvement systems that are embedded in and focused on the 
actual classroom practices of teachers (and not some distal awareness or back- 
ground knowledge); that focus on skill development; and that closely link assess- 
ment with improvement . 3 

CLASS and MTP are not the only systems that reflect this new approach. The 
intent in this more detailed presentation of CLASS -MTP is to illustrate features 
of professional development that are linked to impact on teacher practice and 
student outcomes. Although the focus of the CLASS-MTP model is on teacher- 
child interaction, common features of effective professional development are 
also evident when the focus is on increasing teacher knowledge or on imple- 
mentation of specific curricula. 

Common among all these models of teacher professional development is a highly 
focused target for teacher behavior or knowledge that has a demonstrable link 
to student achievement. This behavior and knowledge can then be the target of 
professional-development activities for teachers and can be directly assessed for 
its impact on these very behaviors or knowledge. In this way, effective professional 
development, whether knowledge-, skill-, or curriculum-focused, reflects a very 
tight coupling, or alignment, between the activities in which teachers engage to 
improve their knowledge and skill and the actual student achievement and social 
behaviors that are the ultimate goals of professional development. In the sections 
that follow, these features are illustrated in considerable detail with regard to one 
approach that focuses on teacher-student interaction as the outcome. 

This report provides background information on the broader challenges of building 
effective professional development for teachers, the theory of action that undergirds 
the CLASS-MTP system, and a description of the CLASS-MTP tools and evidence 
base. The report closes with a discussion of the policy and practice challenges of 
implementing effective systems of teacher professional development on a district or 
statewide level based on the evidence gleaned and lessons learned from this work. 

Among the policy questions addressed are: 

• What are the design and implementation features of effective professional devel- 
opment for teachers? 

• What are the standards for evidence that can be applied to professional develop- 
ment models? 
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• Can proven-effective models also be scalable and cost effective? 

• What are the state and district challenges to adopt or select proven-effective 
models? 

Policy and decision-making recommendations include: 

• Explicit requirements for criteria related to the actual processes by which 
specific professional-development activities are intended to improve teacher 
performance and student achievement. 

• Evidence of impact for any professional-development model should be present 
in order to be eligible for use of public funds. 

• Development and use of guides for selecting professional development for 
district and state leaders that specify explicit criteria for selection of effective 
professional development. 

• Tighter monitoring of the use of public funds for the purposes of providing 
professional development for teachers. 

• Requirements by states and federal agencies that direct; valid assessments of 
teacher performance be included as part of teacher preparation and certification 
systems. Direct assessments actually sample real teaching behaviors as they are 
experienced by students (observations or student surveys) while valid assess- 
ments have demonstrable links to student achievement and other outcomes. 

In the pages that follow, we demonstrate why that time-honored practice, the 
one-day teacher workshop, should die a well-deserved death, and that piling up 
professional-development course credits and advanced degrees has virtually no 
impact on student achievement. We show that one of the most important fac- 
tors in achieving academic progress is a students’ consistent exposure to positive, 
cognitively demanding student-teacher interactions, and that video snippets and 
one-on-one coaching can inspire and transform a teacher’s daily performance. It 
sounds simple, and it is. 
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Background and challenges for teacher 
professional development models 


There are two overarching tasks that a scalable system of effective professional 
development must accomplish. First; the training and support experiences offered 
to teachers must be proven effective for improving practice and student learning. 
Second, to implement a proven-effective model at a statewide or districtwide scale, it 
must be replicable and embedded in systems of incentives, management, and evalu- 
ation that enable high levels of participation and fidelity. Without either of these two 
conditions — a proven-effective model and a system for scaling — ineffective, one- 
time experiences that have little hope of impact will remain the norm. 

Most school leaders know they need to move away from one-time workshops — 
the approach they have used for decades. Experience tells them that professional 
development should be sustained, intensive, and focused on content and practice. 
But where is the supply of proven-effective programs? Where would a school 
leader turn to identify programs that work? 

A recent review of more than 1,300 studies of the impact of teacher professional 
development on K-12 student outcomes found only nine studies that met stan- 
dards for “evidence without reservations” from the What Works Clearinghouse, 
the federal standard for effective-education programs. 4 These nine professional- 
development programs consistently showed moderate effects on student achieve- 
ment, and all involved elementary school teachers. Surprisingly, one of the other 
most commonly deployed approaches to teacher professional development, accu- 
mulating course credits or advancing in terms of degree status (for example, from 
bachelor’s to master’s), has virtually no impact on student outcomes or teacher 
practice. The cupboard, so to speak, is bare. 

There are well- controlled studies, however, that have identified features of profes- 
sional development that do relate to improved teacher practice and student learn- 
ing. These features include: 
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• Specific and up-to-date knowledge of students’ skill targets and progressions 

• Direct support in more effective teacher- student interactions that promote 
student engagement and learning 

• Direct support in effectively implementing curricula 

• Explicit coupling of student and teacher performance-outcome assessments 
(student or teacher) with the professional development activities to which 
teachers are exposed 

Each of these features can be represented in various combinations or focuses, and 
are agnostic with regard to content — that is, they reflect effective professional 
development for math as much as for civics. 

In short, the features of effective professional development are highly targeted on 
relevant knowledge and practice, and there is close alignment between assessment 
(of teacher practice or child outcomes) and professional- development supports 
intended to produce those practices and outcomes. Experimental and highly 
controlled studies suggest that when such targeted, aligned supports are available 
to teachers, student-skill gains can be considerable — on average, 10 percentile 
points on standardized achievement tests and as much as 15 percentile points for 
disadvantaged students. 5 

The starting point for effective professional-development design is a theory that 
explicitly identifies and describes the mechanism through which the profes- 
sional development would have its intended effect (for example, why should this 
specific teacher support work?), and a direct linkage with a system for assessing 
teacher practice (for example, what does good practice look like?) that explicitly 
aligns the professional development with classroom practice. Professional devel- 
opment approaches (such as courses, workshops, coaching, and professional 
learning communities — groups of teachers that gather to analyze and discuss 
their practice) need a clear and strong articulation of why that specific experi- 
ence should change teachers’ practice or student learning (the mechanism, or 
how, of change or improvement). 

Courses, workshops, and coaching must directly produce the skills that can be 
measured as best practices (such as through an existing teacher-observation 
system), and the student outcomes (in other words, achievement) that are the 
intended result. If such mechanisms and intended outcomes are explicitly stated, 
then it becomes possible to evaluate various forms of professional development 
against such claims. In the vast majority of extant professional development, the 
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lack of evidence for or against impacts is in large part due to the absence, vague- 
ness, or inadequacy of the rationale for why they should work in the first place . 6 

Yet even for approaches that do specifically articulate how and why they might 
work, and demonstrate some reasonable impacts on intended outcomes in 
controlled studies, the challenge is scaling up: How can effective programs be 
delivered to large numbers of teachers, either in a district or state? Because even 
for professional-development approaches that work with a few dozen teachers or 
a couple of schools, there is little evidence that they can be implemented in a stan- 
dardized manner across teachers so that many thousands can improve their work 
systematically at district or state levels. 

Over the past decade, our team at the Center for Advanced Study of Teaching and 
Learning, or CASTL, at the University of Virginia has worked on a model of profes- 
sional development to improve teacher-student interactions and in turn increase 
student learning and development. This program of research involved standardized 
descriptions and measures of teachers’ practices that we have used in tens of thou- 
sands of classrooms and experimental studies of different approaches to professional 
development designed to improve teacher practices and interactions. 

We have designed and conducted this work with an interest in a scalable solution to 
supporting teachers in classrooms. Standardized observation of teachers’ classroom 
interactions is the leverage point, based on the logic that if one could observe (across 
many classrooms) practices of teachers proven to contribute to children’s learning 
gains, then those observed behaviors should be the “target” of professional develop- 
ment. That is, our approach to alignment of assessment and professional develop- 
ment has been to identify and measure what teachers do that matters, and design 
and test professional development that produces those behaviors . 7 

The design and evaluation of professional development has been focused on a 
standardized observational assessment of teacher- student interactions devel- 
oped by our CASTL team — the Classroom Assessment Scoring System, or 
CLASS — which predicts student learning and social gains. In many thousands 
of classrooms across grade levels and content areas, higher CLASS ratings predict 
greater student learning, which suggests that CLASS-related behaviors matter . 8 
MyTeachingPartner, or MTP, is a system of coaching and video examples of best 
practices that produce those CLASS behaviors (and higher CLASS ratings) and 
student gains . 9 In our design scheme, this coupling of observation with effective 
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professional development satisfies the first task in building a scalable system of 
effective supports for teachers — a model that works. 

In evaluating professional development, for too long the standard of evidence has 
been some indicator of teacher satisfaction or self-report of learning, usually gath- 
ered using a questionnaire. And too often the published literature includes “evalu- 
ations” of professional development that rely on teachers’ self-reported gains or 
experiences of satisfaction. These evaluations fall woefully short of identifying 
links to teacher practices or student outcomes and fail to include any adequate 
control or comparison groups that would inform inferences about whether the 
professional development actually caused improvement. 

Direct assessments of teachers’ knowledge and skills, and evaluation designs 
that include random assignment experiments or controlled studies of large-scale 
implementation, as a 2007 study indicates, are virtually absent. 10 So when leaders 
and decision makers look for “evidence-based models,” they are left to flail in the 
gap between claims for impact and the needs of their workforce, with the public 
demanding improvement and accountability. As will be evident in what follows, 
and is present in models of professional development that raise student achieve- 
ment through a focus on content knowledge or curriculum implementation, 11 the 
direct assessment of teacher outcomes (for example, knowledge or skills) is one of 
the more critical features. 

The next sections of the paper present the theory behind the CLASS-MTP 
model and a more detailed description of the approach itself. The overall aim is to 
illustrate an approach to professional development in which outcomes (teacher 
behaviors, student learning) are tightly coupled with inputs to teachers (coach- 
ing, video analysis, text) in an aligned system of support. This design approach, in 
which outcomes are tightly coupled with inputs, has been shown effective in our 
team’s work but also in approaches to professional development that are focused 
on content knowledge or curriculum. 
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A professional development focus on teachers' 
classroom interactions with students 


In large-scale testing programs, it is stunningly clear that teachers are the great- 
est source of what students learn and gain as a function of attending school. Yet 
teacher characteristics such as degrees, experience, qualifications, and test perfor- 
mance do not predict student achievement very well, if at all. If teachers indeed 
are so important, what parts of what they do matter? 

Classrooms are powerful settings. Students’ interactions and relationships with 
teachers can either produce or inhibit achievement as well as behavioral and emo- 
tional health. Teachers’ interactions with students stimulate critical thinking and 
convey new knowledge, organize attention and student effort, and motivate, engage 
and support. It is not surprising that measures of teacher-child interactions account 
for much of teachers’ impacts on educational outcomes. A cluster of experimental 
and well-designed descriptive studies show that teacher-child interactions account 
for gains of up to a year’s progress on standardized tests, with even greater effects 
accruing to more disadvantaged children. 12 In fact, findings are almost uniform in 
demonstrating significant and meaningful benefits to children when teacher-child 
interactions are supportive, organized, and cognitively stimulating. 

Unfortunately, the odds are stacked against children getting this kind of educational 
experience. When a thousand children were followed and observed in their early 
childhood and elementary classrooms, only 15 percent of them were found to have 
been exposed to high-quality teacher interactions in multiple grade levels. 13 On aver- 
age, across several thousands of students and K-12 classrooms, children and youth 
are exposed to fairly modest levels of social and emotional supports from teachers 
and rather low levels of instructional quality — levels inconsistent with gap-closing 
classrooms and robust achievement gains. From a different perspective, observations 
in more than 5,000 elementary school classrooms nationwide show that only about 
25 percent of classrooms demonstrate the level and quality of teacher-child interac- 
tions consistent with skill gains for elementary students. 14 

And in middle and high school, adolescents routinely describe feeling disengaged 
in the classroom and alienated from school, a gap between teacher and learner that 
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clearly contributes to dropping out and underperformance. Yet engagement and 
intrinsic motivation become pivotal in adolescence, as students at this age have 
the means to not only withdraw energy from educational pursuits but to drop out 
altogether. Engagement in school begins to decline by fifth grade and by entry 
into high school it is so pronounced that half of high school students report that 
they do not take their school or their studies seriously. 15 But despite the generally 
dismal picture, nearly every teenager can describe, with enthusiasm and passion, 
a relationship with a teacher that they felt was meaningful and important to them, 
often with considerable evidence to back up those claims. 16 The most pressing 
questions are, how can such teachers and teaching be replicated on a very large 
scale, and what is the role of professional development in achieving such results? 

Improving the quality and impact of teacher-student interactions within the class- 
room depends upon a solid understanding of just what is effective teaching. 


Improving the 
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depends 


Our team at CASTL developed the Teaching Through Interactions, or TTI, frame- 
work over the past decade to organize and describe the wide range of teacher-stu- 
dent interactions associated with academic and social development. This conceptual 
framework focuses on teacher behavior and what students actually experience in 
three broad domains of teacher- student interaction: emotional support, classroom 
organization, and instructional support. Within each broad domain are specific 
dimensions of teacher-child interaction that further specify the forms of teacher 
behavior that matter, (see Figure 1 ) Importantly, we posit that these areas of interac- 
tion are important for students from preschool age through high school — a hypoth- 
esis that has received considerable support in several studies. 17 
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As shown in Figure 1, the broad domain of emotional supports is defined in terms 
of three dimensions: positive classroom climate, teacher sensitivity, and regard for 
student perspectives. Each of these dimensions is then specifically defined by a set of 
categories of behavior that yield observable teacher behaviors and interactions. For 
example, classroom climate includes observable behaviors such as the frequency and 
quality of teachers’ emotional communications with students (specified in terms 
of vocal tone and positive verbal feedback) as well as the degree to which students 
interact positively with one another, (see Table 1 for further detail) 


TABLE 1 

Teaching Through Interactions Framework Dimensions 

Observable categories of teacher-student communication 

Domain Dimension Description 

Reflects the overall emotional tone of the classroom and the connection between teach- 
ers and students. Considers the warmth and respect displayed in teachers' and students' 
interactions with one another, as well as the degree to which they display enjoyment and 
enthusiasm during learning activities. 

Encompasses teachers' responsiveness to students' needs and awareness of students' level of 
Teacher sensitivity academic and emotional functioning.The highly sensitive teacher helps students see adults as a 
resource and creates an environment in which students feel safe and free to explore and learn. 

The degree to which the teacher's interactions with students and classroom activities place an 
Regard for student emphasis on students'interests, motivations, and points of view, rather than being very teacher- 
perspectives driven. This may be demonstrated by teachers' flexibility within activities and respect for students' 

autonomy to participate in and initiate activities. 


Emotional 

support 


Classroom climate 


Classroom 

organization 

Behavior 

management 

Encompasses teachers'abiiity to use effective methods to prevent and redirect misbehavior, by 
presenting clear behavioral expectations and minimizing time spent on behavioral issues. 


Productivity 

Considers how well teachers manage instructional time and routines so that students have the 
maximum number of opportunity to learn. Not related to the quality of instruction, but rather 
teachers' efficiency. 


The degree to which teachers maximize students'engagement and ability to learn by provid- 
Instructional ing interesting activities, instruction, centers, and materials. Considers the manner in which the 

learning formats teacher facilitates activities so that students have opportunities to experience, perceive, explore, 

and use materials. 


Instructional Concept The degree to which instructional discussions and activities promote students' higher-order 

support development thinking skills. 


Quality of feedback 


Considers teachers' provision of feedback focused on expanding learning and understanding 
(formative evaluation), not correctness or the end product (summative evaluation). 


Source: Robert C. Pianta 


Language modeling 


The quality and amount of teachers' use of language-stimulation and language-facilitation tech- 
niques during individual, small-group, and large-group interactions with children. Components 
of high-quality language modeling include self and parallel talk, open-ended questions, repeti- 
tion, expansion/extension, and use of advanced language. 
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Classroom observations provide the most direct way to measure the interactions 
outlined in the Teaching Through Interactions framework. Therefore; the primary 
way in which the TTI framework has been applied is through the standardized 
observation protocol 18 of CLASS. CLASS is a standardized observation and 
assessment system with versions for prekindergarten, elementary, and secondary 
classrooms that yields quantitative ratings (on a one-to-seven scale) for each of 
the dimensions of teacher- student interactions in the TTI framework. Analysis 
of CLASS ratings across many thousands of classrooms confirms that the three 
broad domains of interaction — emotional support, organizational support, and 
instructional support — are reasonably well-fitting descriptors of teacher-student 
interactions across grade levels. 

Assigning quantitative ratings to something as complicated as teacher-student 
interaction in a classroom requires a well-trained eye watching for enough time 
to make a judgment. To use the CLASS reliably, for live or video observations, a 
set of three to four 20-minute cycles of classroom observation is recommended. 
And observers must undergo a training protocol to become calibrated with 
“gold-standard” coders. 

This requirement for reliability — for an observer to demonstrate agreement with 
a standard and for the instrument itself to be sufficiently specified to support such 
agreement — is a fundamental property of any assessment tool and a critical compo- 
nent of observation. Most of the widely used approaches to observation, including 
district-developed principal observations as well as some commonly used “rubrics,” 
fail to describe their reliability and typically do not require users to demonstrate 
such agreement with a standard. And as districts often do not standardize their 
approaches to observation, do not use reliable metrics, do not provide adequate 
training and support to observers to observe reliably, then ratings obtained from 
such procedures are fraught with unknown and likely large amounts of errors, and 
conclusions drawn from such scores or ratings maybe in error as well. 

In studies that have used the CLASS system and followed these standard proto- 
cols for training and observation, there is consistent evidence that CLASS ratings 
capture features of teachers’ behavior that are consistent from day to day, across 
times of the year, or the content area the teacher is teaching. 19 In other words, 
ratings assigned to a teacher for “Quality of Feedback” in October and April will 
be quite consistent. Or ratings for Ms. Smith’s interactions when she is teaching 
language arts will be similar to those when she is teaching history. 
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Furthermore; the quality of teacher- student interaction assessed by CLASS pre- 
dicts students’ social and academic development. In more than 700 preschool 
and kindergarten classrooms in an 1 1-state study higher levels of instructional 
support predicted children’s language, literacy, and math gains, while emotional 
support predicted social skills, with effect sizes ranging from small to moderate, 
roughly accounting for changes of between 5 and 15 percentile points on standard- 
ized tests. 20 In studies involving several thousand kindergarten through fifth-grade 
classes, higher ratings on instructional and emotional support accounted for greater 
gains on standardized assessments of academic achievement and better social adjust- 
ment, even adjusting for teacher-, program-, and family-selection factors. 

In middle and high school classrooms, a combination of all three domains of 
teacher-student interaction (emotional, organizational, and instructional) scores 
accounted for improvements in student engagement and noticeable gains in stu- 
dents’ scores on the high-stakes state standards test. In fact, the difference between 
gains for students enrolled in the highest- and lowest-rated classrooms was about 
25 percentile points. 21 Interactions between teachers and students matter; they are 
not the only things that matter in education, but they matter, and therefore could 
be a target for professional development. 

In sum, the TTI framework and CLASS observational assessment are valuable 
tools for understanding and measuring teacher-child interactions. The framework 
and assessment tool not only define effective teaching but also quantify it, from 
high to low. Observational data gathered using the CLASS framework provide 
valuable information for a number of purposes, including accountability-driven 
assessments of teacher quality, facilitation of professional development, and devel- 
opment of theory about how classrooms influence students in positive or negative 
ways. Teachers routinely report the value of the CLASS system as a “road map” for 
how to improve their teaching, or say that CLASS validates and provides a struc- 
ture for their own explanations, interpretations, and analysis of their practice. 

What we have learned, we think, is the value for teachers of having an explicit and 
common language and lens for their practice. Having developed a definition and 
measure for effective teaching, we then set out to use these tools directly, overtly, 
and explicitly to design and evaluate a set of professional-development resources. 
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Improving teacher-student 
interactions: MyTeachingPartner 


The MyTeachingPartner, or MTP, system of professional-development supports, 
which our team at CASTL developed in the late 1990s, combines the features of 
emerging and promising professional development presented in the paper’s first 
section with the focus on teacher- student interaction in the second to provide 
teachers with professional-development inputs that include: 

• Extensive opportunities to observe effective teacher-student interaction through 
analysis and viewing of multiple video examples 

• Skills training in identifying effective and ineffective instructional, linguistic, and 
social responses to students’ cues, and identifying the behavioral sequences con- 
necting teacher responses to students’ engagement and skills 

• Repeated opportunities for individualized feedback on and analysis of one’s own 
interactions with students 

When the MTP system of professional-development supports is linked to CLASS 
observations and definitions of effective practice, a direct path can be traced 
from inputs to teachers (professional development) to teacher inputs to chil- 
dren (teacher-student interactions) to children’s skill gains. Thus we have closely 
attended to the need to identify a mechanism (the how and why it works) for these 
approaches to professional development. The suite of MTP professional-develop- 
ment supports contains three specific resources linked to the inputs above: 

• A video library of annotated examples of best practice 

• Web-mediated individualized coaching 

• A college course focused on the CLASS system 

All three rely on the CLASS system as the primary way to observe and define 
effective practice. This section of the paper describes each of these approaches to 
professional development and the results related to improving teachers’ effective- 
ness and children’s learning. 
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Importantly, the MTP system of professional development is not focused on 
teachers’ content knowledge, and these resources are not content- or grade-level 
specific. This is because the system is conceptualized to operate on features of 
teacher-child interactions that are valuable and applicable across content areas 
(an assumption supported by the evidence to date) and grades. For the pur- 
poses of scalability in school or district, the importance of all teachers having a 
common lens for effective instruction was considered a priority in the design of 
the CLASS and TTI framework. 


The MTP video library 

The MTP video library — more than 400 one- to two-minute video clips of 
teachers’ effective interactions with students from pre-K to high school — gives 
teachers an opportunity to observe other teachers’ effective interactions as they 
implement a wide range of instructional activities in various contexts. These 
examples are from schools all across the country and represent a wide range of 
children, teachers, and educational approaches. They also include examples of 
instruction in a variety of content areas and with children who represent diverse 
language, home, and ethnic backgrounds. 

Each clip is accompanied by a detailed and specific description of the behavior 
that explains why it is effective in fostering student learning and development. The 
videos are organized (and searchable) by CLASS dimensions, and each one includes 
an explicit, behavioral description of what is happening in the footage that reflects a 
high-quality teacher-child interaction. Teachers are often isolated within their own 
classrooms and rarely have the opportunity to see authentic, real-time examples of 
effective interactions and teaching. The idea behind the MTP video library was to 
provide opportunities for teachers to leam more about what types of interactions 
were important for student learning (knowing) and understand exactly what these 
interactions look like when enacted by others (seeing). 


MTP coaching 

MTP coaching involves observation-based analysis and feedback enacted through 
a regular cycle of web-mediated interaction that takes place both in real time 
exchanges between the teacher and coach and opportunities for the teacher and 
coach to interact without both having to be online at the same time, (see Figure 2). 
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Every two weeks over the course of the academic year, using a simple flipcam setup 
on a tripod in their classroom, teachers videotape their own instruction in literacy, 
language, and social skills and send this footage to their coach. 

The consultant edits the tape into three segments that focus on a specific CLASS 
dimension (for example, concept development) and on: 

• Identification of instances of effective behavior 

• Analysis of alternatives 

• Instruction 

The segments and feedback (called “prompts”) are posted to a secure website where 
teachers view the segments and comments and then respond to the prompts. 

The prompts focus attention on specific aspects of teacher-child interaction, 
such as “What did you do here to help ‘J’ pay attention, and what did he do in 
response?” and include links to clips of effective practice from the video library. 


figure 2 

The MyTeachingPartner Coaching Style 

Analyzing classroom behavior ^ Teacher records 

through videos and feedback Step classroom 

video 


Step 

Summary and 
action plan inform 
next cycle 


Teacher and coach Teacher reviews 

Step discuss prompts Step video and responds 

and practice to prompts 

Source: Robert C. Pianta 
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Both the response to coach feedback online and the conversations during con- 
ferences (these can be phone calls, or Skype-type teleconferences) are set up to 
encourage teachers’ analysis of and reflection on their own interactions. Namely, 
teachers are challenged to analyze how their own actions are linked to certain 
student cues and responses, and vice versa. The coach also prompts the teacher 
to take the next step and plan and create future learning opportunities that will 
replicate successful interactions, or try out new ways of engaging children in 
learning through interactions. 

This MTP coaching cycle is spread over two weeks and repeated throughout the 
school year, gradually deepening the reflective analysis around interactions that 
are more difficult to detect and enact. All together, these loops of videotaping, 
feedback, and conferencing encourage teachers to observe, enact, and understand 
the connections between their behavior in the classroom and student learning. 
MTP coaching increases teachers’ knowledge to define — and skills to observe — 
interactions with students, and their awareness of their impacts on student moti- 
vation, teacher- student relationships, and student learning. 

Once developed, we tested these MTP professional-development resources in a 
randomized controlled trial with more than 240 state-funded pre-K teachers. All 
teachers had access to the MTP video library, and some had access to MTP coach- 
ing as well. Compared with the video-only teachers, teachers assigned to coaching 
made significant gains in reading and responding to students’ cues, using a variety 
of formats to actively engage children in instruction, and intentionally stimulating 
language development. 

These impacts were roughly on the order of one scale point on the CLASS-rating 
system — for example, moving from a rating of “3” to “4” in one academic year. 
And, in classrooms where 100 percent of the children were from economically dis- 
advantaged families, there were remarkable differences in teachers’ rates of change 
in sensitivity and responsiveness, and facilitating engagement and enthusiasm 
in learning. Teachers receiving coaching increased the quality of their interac- 
tions roughly 1.5 rating points on the CLASS scale. In addition, comparisons of 
children’s achievement, learning, and social skills indicated that children in MTP- 
coached classrooms made greater gains in receptive vocabulary, task orientation, 
and prosocial assertiveness, scoring approximately 4 to 5 percentile points higher 
than children in the other conditions on standardized tests. 

Clearly, the evidence was in favor of the coaching condition. There were hints 
in follow-up analyses, however, to suggest that teachers who had access only to 


17 Center for American Progress | Teaching Children Well 


video clips and made regular use of them were observed to be more sensitive and 
responsive to children’s needs, more proactive and effective at managing behav- 
ior, and more skilled at maximizing childrens learning time. In short, for some 
teachers just watching examples of effective teaching on the video library made a 
difference in improving their practices. And those teachers who spent more time 
watching video examples of best practices and reading the accompanying detailed, 
CLASS-focused descriptions online also had children who experienced greater 
gains in social development during the pre-Kyear. Teachers who spent more time 
watching and analyzing their own classroom video footage online as part of coach- 
ing also had children who experienced greater gains in vocabulary development 
during the pre-K year, with gains equal to approximately 5 percentile points. 

Collectively, these findings suggest that the act of watching oneself and others 
interact with children in a classroom in and of itself may contribute to changes in 
instruction and children’s learning. Additionally, focus groups with teachers at the 
end of the project indicated that many of the teachers, particularly those receiving 
coaching, felt that they could have taken better advantage of reflection work with 
their coach if they had a greater understanding of what constituted effective teacher- 
child interactions at the beginning of the year. They needed more knowledge about 
these practices and more time to see what they looked like in classrooms. 

We recently scaled-up the MTP coaching model to 450 pre-K teachers in 15 sites 
served by local consultants 22 trained and supported by our team (all using standard- 
ized procedures). When evaluated in a randomized controlled trial, the benefits of 
MTP coaching were confirmed. Teachers who received MTP coaching improved 
in nearly every feature of interaction assessed by CLASS, particularly the dimen- 
sions of instructional support, with effects averaging approximately one point on 
the CLASS dimensions. This study not only replicated the value of MTP coach- 
ing for teachers, but demonstrated that the model of coaching can be replicated 
and extended through standardized training and support protocols. Importantly, 
although a one-point magnitude difference on the CLASS one-to-seven rating scale 
may seem small, recall that prior research demonstrated that a one-point difference 
in the quality of teachers’ interactions, particularly in the instructional domain, was 
associated with increased achievement outcomes for children. 

We then sought to examine the applicability and replication of MTP for second- 
ary teachers and classrooms. 23 When we used and evaluated MTP coaching in a 
randomized controlled study of teachers in middle and high school classrooms 
across four academic content areas, the results provided some of the best evidence 
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to date that professional development targeting student-teacher interaction can 
alter not only teacher behavior but also produce student-achievement gains on 
high-stakes tests. The MTP-Secondary coaching program produced observable 
changes in teacher behavior that had been previously linked to student achieve- 
ment and led to gains in student achievement in the following year. 

These effects were meaningful for teachers, students, and schools. On average, 
gains were equivalent to increasing the achievement status of every student in a 
class taught by a teacher who had been in MTP-Secondary from the 50th to the 
59th percentile, and to preventing one student from otherwise failing the end of 
year state assessment . These gains were the same across content areas as well. The 
average gains per student across content domains, if accrued across all students in 
a grade or school, could easily push a school above annual-progress thresholds. 

The results are also striking because MTP coaching appears cost-effective as we 
begin to evaluate impact across years and students relative to time and expense. 
MTP teachers participated in a one-day in-service session prior to the coaching 
year, and on average engaged in nine coaching cycles, requiring approximately 30 
minutes each. In terms of total teacher time, MTP coaching required about 20 
hours per teacher across two full years (one year of coaching, two years of com- 
pleting assessments). 

The full cost was $3,500 per teacher for coaching and $200 for video equipment 
over this period. One can imagine that the teacher-coach cost might well offset or 
be covered in-kind by existing teacher training/ supervision programs in schools. 
Even assuming that MTP coaching costs were fully realized, they still compare 
favorably to the annual $3,000 to $9,000 typically spent on teacher in-service 
training. Looked at another way, for a cost of less than $165 per participating 
student, MTP coaching produced a boost of 9 percentile points in achievement 
and on average would prevent one course failure in each classroom in which it 
was implemented. The finding that the effects carry over to a second year, during 
which consultation was not offered, raises the possibility that the effects will be 
more durable, further increasing the potential cost effectiveness of the program. 

Notably, in spite of the obvious importance of secondary school success for stu- 
dents, and evidence of the current struggles of secondary schools, MTP coaching 
appears to be one of the only professional-development programs to demonstrate 
efficacy in improving teacher effectiveness and student achievement when evalu- 
ated in a randomized controlled trial. And in the secondary school evaluation 
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study^ follow-up analyses suggested that the impacts on student learning were 
indeed due to the specific qualities of student-teacher interaction that were the 
MTP coaching focus, as we had also found in the pre-K study. 


The MTP Course 

The third professional development resource for teachers is the MTP Course, 
“Support of Language and Literacy Development Through Effective Teacher- 
Child Interactions and Relationships.” This three-credit college course focuses on 
improving teachers’ knowledge of effective interactions, their skills in identifying 
effective interactions, and applying those skills to their own classrooms. 24 Like the 
other MTP professional development resources, it was initially developed for and 
evaluated in pre-K classrooms by our CASTL team as part of the Department of 
Education- funded National Center for Research on Early Childhood Education. 

The course targets two major clusters ofbelief, knowledge and skill — the first 
focuses on effective teacher-child interactions and the second specifically on 
teaching of literacy and language in early-childhood classrooms. The course uses 
the TTI framework for background knowledge, and the CLASS system to make 
explicit links between behavioral actions and intended consequences for children. 
For example, when learning about quality of feedback, teachers are encouraged 
to watch and analyze videos that highlighted the ways in which specific teacher 
actions lead to children spending more or less time thinking about and analyzing 
a problem. The course also targets teachers’ skills in detecting effective teacher- 
child interactions though video analysis. We hypothesize that it is not sufficient 
for teachers to be able to gain knowledge about effective interactions; they need 
actual skills involving identification of effective interactions with a high degree 
of specificity in order to be most likely to transfer the coursework into real-life 
changes in their instructional practice. Therefore, a primary emphasis in the 
course is analysis of videotapes from real classrooms to develop skills of identify- 
ing effective (and ineffective) interactions, and articulating specific behavioral 
evidence to support these judgments. 

This course was delivered in 14 three-hour sessions through collaborations with 
local colleges and universities in 10 sites across the country. There were between 
5 and 15 teachers in each course section. Fifteen instructors were provided with 
standardized manuals that included PowerPoint presentations, videos, and written 
assignments for each course section. Instructors attended a weeklong training 
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session and were provided with ongoing support by our staff’ including weekly 
phone calls from course developers. Videotape coding of course sections indi- 
cated that instructors were teaching the course as we had trained them to do, using 
the materials and session guides according to the plan. Not only were we able to 
study the impacts of the course in a controlled experiment, we also evaluated the 
extent to which these impacts were achievable across a wide range of settings and 
institutions; thus, this was also a study of scaling-up. 

Compared with teachers in the control condition, those in the course condition 
reported more intentional teaching beliefs and demonstrated greater knowledge 
of, and skills in, detecting effective teacher-child interactions. 25 Teachers in the 
course condition also reported stronger beliefs about the importance of teaching 
children early literacy and language skills, and demonstrated greater knowledge 
about these skills. Those who took the course demonstrated more effective emo- 
tional and instructional practices in interactions with children. 

The magnitude of impacts of the course on teachers’ observed classroom practices 
was between one-half and one scale point on the CLASS metric. Importantly, 
teachers’ skills in detecting effective interactions partially accounted for the effects 
of the course on teachers’ actual practices. What has been even more encouraging 
is that we see continued changes in knowledge and in quality of teacher-student 
interactions in the year following exposure to the course, suggesting some degree 
of stable improvement. 

In sum, results from the evaluation indicate the MTP course produces significant 
changes in teachers’ knowledge of skill targets for children’s learning outcomes, 
knowledge of teacher-child interactions, skills in identifying interactive cues and 
behaviors, and improvements in instructional practices. 

Overall, the MTP system of professional development — a course, video library, 
and web-based coaching through which specific professional- development activi- 
ties for teachers focus directly on a measure of teacher-student interaction that 
relates to student achievement — appears not only beneficial for changing teacher- 
student interactions and improving student learning and development from pre-K 
to grade 12, but also appears replicable and scalable. 26 The next section of this 
paper outlines some of the implications of these results for systems of accountabil- 
ity, evaluation, and human- capital management and development. 
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Effective teacher professional 
development: Implications for teacher 
preparation, credentialing, and scale-up 


This section explores broader implications of this promising approach to profes- 
sional development. To the extent that the MTP resources for teachers actually 
produce gains in teachers’ skills that produce better results for learners there are 
three key avenues for further exploration: teacher preparation, teacher certifica- 
tion, and teacher-evaluation systems. 

First, it seems critical to build this type of competency-based opportunity for 
professional preparation “back” into the “pre-service” sector. That is, if effective 
professional development has been identified and linked to assessments of teacher 
competence, then it seems reasonable to suggest that those professional-develop- 
ment models and associated assessments could serve as focal opportunities and 
metrics in teacher preparation. As just one illustration, could the MTP course be 
used in teacher preparation, and could it be used widely across many programs ? 
Could it be expanded or extended to upper grades? What factors would facilitate 
adoption by teacher-preparation programs of a course that is both scalable and 
proven to improve teacher practice? 

Second, the existence of proven-effective models for professional development and 
quantitative metrics for effective practice could be linked to states’ credentialing and 
certification systems. Teachers could receive credentials or certification to practice 
on the basis of their participation in effective professional development and demon- 
stration of competence. That is, could one imagine that teacher credentials would 
in part be awarded when teachers were observed to demonstrate levels of effective 
practice that exceeded a certain threshold on a validated observational measure? 

Moreover, if a particular form of professional development were proven effective for 
improving teacher and student performance (for example, MTP coaching), would it 
be possible for teachers participating in such professional-development opportuni- 
ties to be credentialed based on accumulation of demonstrated competencies? 
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In software engineering, career ladders are based on a series of certification tests of 
various competencies in programming languages, such as Java script and HTML, 
which have been identified as key markers for performance in the profession. 

These certification tests function as a de facto credentialing system, as companies 
hire on the basis of test performance/ certifications. Is it not conceivable that a 
similar system of valid competency tests and associated professional development 
and training opportunities could operate in teacher preparation and certification? 

To re-emphasize a point made earlier, such performance tests and preparation 
experiences must pass high thresholds for evidence of reliability (low levels of 
error), validity (they really produce results), and scalability. Although teacher 
certification and preparation is not structured like software engineering yet, can 
we imagine and design toward a future in which a progression of increasingly com- 
plicated and sophisticated professional skills and competence has been specified 
and is the focus of training, evaluation, and certification? 

One can imagine these professional- development and observational-assessment 
systems being integrated into larger human-capital management systems, or 
connected to incentives such as merit pay. With the emphasis on assessment of 
teacher performance and the importance of developing systems that produce 
effective teaching, the availability and use of standardized assessments of teacher 
performance and aligned, effective professional development is critical for the suc- 
cess of these systems with regard to improvement of student learning. 

A major implication of this work is that teachers’ effective practices in classrooms 
can be reliably observed using standardized protocols, predict student learning 
gains (even on state standards tests), and can be scaled up in very large systems 
(for example, Head Start’s 50,000 classrooms) using live observers and train-the- 
trainer models of web-based instruction and video scoring. Thus it is possible to 
meet the demands of Race to the Top and other recent initiatives to better assess 
and evaluate teacher performance using observation as one of the metrics. 

Such feasibility means that measures of effective practice can be inserted into 
accountability and monitoring frameworks, expanded into performance-improve- 
ment and program- development models in schools or districts, and linked to effec- 
tive professional development designed to improve the very behaviors on which 
teachers will be monitored, evaluated, and assessed. This potential for a linked, 
or coupled, system of monitoring, assessment, and improvement based on actual 
competencies could then be connected to a set of performance plans, rewards, and 
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perhaps even certification structures that in the end tie back to actual competence in 
the classroom — rather than seat time or exposure, as is now the case. 

As just one example of the importance of standardized-observational assess- 
ments of teacher- child interactions, consider the fact that in teacher-preparation 
programs, well over 95 percent of the nation’s teachers-in-training are observed 
during their student-teaching placements, ostensibly to gauge their skill level with 
regard to competencies deemed desirable or even necessary by their higher educa- 
tion program or state-licensure systems. Then consider that in fewer than 15 per- 
cent of these observations is there evidence for the reliability of the instrument, 
much less validity. 27 Once they move into roles as teachers, the evidence is that 
virtually no teachers are observed using a validated tool. Furthermore, all teach- 
ers in training receive supervision of their practice, with little to no evidence that 
these supervision models actually improve these teacher-candidates’ practices. 

Thus, despite the fact that interactions in a setting intended to promote positive 
development are the criterion around which competence will be judged for this 
group of professionals, there is no agreed-upon operationalized standard by which 
performance can be judged reliably and validly, and no proven-effective model 
for improving practice. Importantly, every state, district, and teacher-preparation 
program has standards for teachers’ behavior in the classroom — management of 
classroom behavior, providing students with feedback about their learning — but 
in 85 percent or more of the occasions on which a teacher is observed with regard 
to these standards, there is no agreed-upon operational definition of how teachers 
should behave in the classroom and what distinguishes adequate from inadequate 
performance. There are even fewer examples of professional-development models 
that are effective for improving the very performance standards that various enti- 
ties deem so important and essential. 

Principals and supervisors engage in widespread use of observations in class- 
rooms, from drive-by drop-ins to more extended analysis of teachers’ behaviors. 
Nearly all consumers and producers of education agree that observation of teach- 
ing is an important part of evaluation and of professional development. Toward 
this end, observation is noted as a foundational component of multiple-indicator 
rubrics for gauging teacher quality that have sprung up in the latest round of 
standards-based education reform efforts, such as Race to the Top. But the vast 
majority of observations conducted in U.S. schools are not standardized, the 
scores obtained are not reliable, and they don’t predict students’ learning gains. 
This lack of a common set of indicators, evidence of reliability across observers, 
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or common procedures — to say nothing of the lack of evidence for validity — all 
undermine the extent to which they support inferences that could be useful either 
in policy or in work with individual teachers. 

A common question regarding the impacts of MTP concerns cost — particularly the 
expense of coaching, which we address here. In so doing, it is important, however, to 
consider costs not only just for coaching per se, but also in the context of the other, 
less-costly but effective components of the MTP suite of professional-development 
resources (course and video library). Moreover, costs are perhaps better understood 
when proven-effective professional-development strategies are integrated within 
human-capital management systems. Such systems might include: 

• A universally applied initial screening/monitoring assessment (CLASS 
observations) 

• An equally universally-applied low-cost professional-development support 
system (video library subscriptions) 

• A decision tree for triaging teachers into more intensive supports (coaching or a 
course) based on need and performance data 

Thus a coordinated, aligned system of assessments and supports of varying 
intensity can allow for calibration of costs in relation to need, so that higher-cost 
resources are applied only when needed. 

And cost has to be estimated in relation to existing expenses devoted to similar 
activities (observation, coaching, courses). The costs of a teacher-coach might well 
be already covered, for example, as many schools employ coaches, instructional- 
support personnel, and mentors for early career teachers at considerable expense. 
Thus, the labor expenses are not the problem; it is that these professionals typi- 
cally are not implementing a standard or effective model. 

Similarly, nearly every teacher-preparation program in the country already pays for 
instructors to teach courses in how to interact effectively with children. Yet there 
is no evidence those courses have an impact on relevant knowledge or skill. Again, 
the labor for implementation is already accounted for in existing budgets; what’s 
missing is a set of effective models to implement. 
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In fact ; because the current cost per teacher for professional development; esti- 
mated to be between $3 ; 000 and $9,000 annually, produces little to no impact 
on practice or student learning, the real task is to shift existing funds to high- 
fidelity implementation of proven- effective programs. This cost-shifting strategy 
is borne out by the pre-K scale-up study of MTP coaching using local personnel 
as coaches. Although teachers could access MTP coaching through web-based 
interactions with coaches located remotely in which all costs are “new,” it is now 
clear that local coaches can be trained to deliver MTP coaching with similar, if not 
greater, impacts on practice. 

These results open the potential for existing personnel involved in supervision or 
coaching (school districts already employ many of these individuals at consider- 
able expense) to be trained to use an MTP coaching model (or other proven- 
effective model) in the context of their existing roles. Thus existing resources can 
be readily redeployed to implement proven- effective professional-development 
models at considerably lower costs. 
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Conclusion 


There is widespread acknowledgement that the production of effective teaching and 
teachers is perhaps the critical component of education reform and innovation for 
improvement of student learning. This aim requires a serious investment of time, 
rigor, and evaluation to produce professional-development programs that actually 
work. This paper has described the conceptual, operational, and evidentiary basis 
of one such model, the CLASS-based MyTeachingPartner suite of professional- 
development resources, ha so doing, the paper presents a proof of concept and an 
illustration of the features of a model with some evidence of success. 

The consideration of costs for proven-effective professional development must be 
made in light of policies and incentives for delivery of proven-effective models. 

In this regard, states could be far more active in requiring evidence of impacts 
from professional-development providers, higher education settings, and product 
vendors before any of their resources could be posted on an “approved” list. In 
fact, states could be far more prescriptive in giving directions to localities, where 
much of the choice regarding the allocation of professional-development funds 
resides. If states want localities to retain control over expenditures, then they must 
ramp up considerably the supports and training available to local decision-makers 
so that they choose wisely from effective programs. Just requiring localities to 
spend money on “research-based” or “evidence-based” programs is not enough in 
a world in which most anything can be labeled as such. 

There are other models for which evidence of impacts is also available. These 
models share several common elements: a strong conceptual framework that 
provides the basis for expecting impacts; clear alignment and linkages that con- 
nect input to teachers, to inputs from teachers to children, to children’s learning; 
and strong metrics and standardized protocols that support standardization and 
high fidelity of implementation. Although costly on their own, these models 
are no more costly (and in fact less so) than the wasteful spending now occur- 
ring on most professional development. And with emerging policy frameworks 
that provide structural and systemic supports for scale-up, the central question 
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is how to shift resources and practices from the current, ineffective models to 
those with promise and evidence, while at the same time investing in developing 
a much greater supply of proven-effective models. The emerging track record of 
the CLASS-MTP model, in which scalable observational assessment is linked to 
proven- effective approaches to improve practice and student learning, suggests 
considerable promise for progress. 

Four policy-relevant recommendations follow. 


Explicit requirements for criteria related to mechanism and 
evidence of impact should be present in any professional- 
development model eligible for use of public funds 

Public dollars for the improvement of teaching are precious and scarce, and 
should be used well. It should be required that vendors of professional develop- 
ment be highly explicit about the mechanisms of impact (how and why it works) 
and evidence of impacts (from well-controlled studies) for their materials and 
procedures to be purchased using public funds. 


Guides for selecting professional development for district and 
state leaders specifying explicit criteria for selection of effective 
methods should be developed and used 

District and state leaders are overwhelmed with information and pressured by 
constituents for results. This is not a situation conducive to good decision making, 
particularly when coupled with vague or poorly described (and sometimes mislead- 
ing) information provided by developers. Decision-makers need tools to implement 
policy well — in this case, a set of explicit guides, if not even lists, of professional- 
development features or models that can drive their decision-making toward the 
selection and implementation of models that are more likely to be effective. 


Tighter monitoring of the use of public funds for the purposes of 
providing professional development for teachers should be mandated 


Districts and states should be required to more explicitly justify (a priori) their use 
of public funds for the purposes of teacher professional development. 
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States and federal agencies should require that direct, valid 
assessments of teacher performance be included as part of teacher 
preparation and certification systems 

Standards for teacher performance are not enough. States, districts, and teacher- 
preparation programs should be required to use direct assessments of teacher 
knowledge and skill that have demonstrated empirical links to objective indica- 
tors of student learning, development, and achievement. Requiring that teachers 
demonstrate valid knowledge and skill in order to receive certification will drive 
preparation programs to produce those skills. In the absence of performance 
measures linked to certification, not even the “best” or most “rigorous” list of 
standards will produce change. 
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